MVT Overview

Overview

Multivariate testing (MVT) is a well-known statistical technique for testing hypotheses on complex systems. MVT allows you to test whether changes made to Algonomy recommendations and/or to your website will have a positive or negative impact on your Key Performance Indicators (KPIs), using live customer traffic.

MVT allows retailers to compare different settings and configurations to statistically prove which condition more accurately drives the desired customer behavior.

Setting up tests should be done with a clear idea of what is being tested and a statistical understanding of significance.

You can create one or more tests. Each test will at a minimum have one treatment and one control. You designate a desired percentage of traffic for each treatment. Customers who are placed in your test are assigned to the control or one of the treatments according to these desired percentages.

Customers are placed in one test at a time; you can have multiple tests running simultaneously, as long as the traffic percentages across the tests add up to no more than 100% of traffic. The concurrent option allows the customer to create tests that can run in parallel without affecting any of the merchant's traffic.

Multivariate Testing

When you set up a multivariate test, you split your customers into a control population and one or more treatment populations. You can then compare these populations to determine whether the changes being tested had an impact on KPIs such as:

  • revenue per session

  • conversion rate

  • average order value

  • click-through rate

You can find more details on the concept of MVT here:http://en.wikipedia.org/wiki/Multivariate_testing

We currently support the ability to MVT the following items:

  1. Test merchandising rules within recommendations

  2. Test the location of one or more recommendation placements on one or more pages

  3. Test the number of placements on a page

  4. Test different strategy configurations on one or more placements

  5. Test recommendations against no recommendations

  6. Test creative elements, like whether inserting a "add to cart" button increases conversion

  7. Test Discover global and personalized weights and settings.

In coordination with your Algonomy team, additional page elements can also be tested.

Treatment Population

This assignment of a customer to a treatment is done by mapping the session id or user id to the test and treatment in a reproducible way. This way, as long as the user's ID remains the same, so does their treatment assignment, regardless of whether they allow cookies, clear cookies, etc. Consult with your Algonomy contact to change the identifier used for treatment assignment, or to discuss the best choice of identifier.

The MVT system measures the behavior (views, clicks, and purchases) of your customers in each of these buckets (Control, A1, B1, B2) and reports their aggregate performance.

You can create many tests and treatments at a given time. The only restriction is that the sum of all traffic assigned to these treatments does not exceed 80%. (This is because Algonomy Omnichannel Personalization reserves 20% of traffic for the control.) You can assign the required traffic percentage to a test and then distribute that traffic between different treatments.

In order to ensure your tests are accurate, we recommend that each MVT test runs a minimum of two weeks. This will give your tests ample time to normalize.

Example

Let us say you have two MVTs named  MVT1 and MVT2. MVT1 has treatment A1 with 20% traffic assigned to it, and a control treatment C1 with 20% of traffic. MVT2 has two treatments A2 with 10% of traffic, B2 with 20% of traffic, and a control treatment C2 with 15% of traffic. Thus, MVT1 has 40% of traffic overall, and MVT2 has 45% of traffic (10% for A2, 20% for B2, and 15% for C2).

When a customer arrives at your website, they have a 40% chance of being in MVT1, a 45% chance of being in MVT2, and a 15% chance of being excluded from any test. Within MVT1, they have an equal chance of being in treatment A1 or in the control C1. Within MVT2, they may be put into A2, B2 or C2, weighted in such a way as to achieve the desired traffic percentages.

Below only applies to the legacy, cookie-based treatment assignment, need to move this off to some internal documentation as part of moving to deterministic as the best practice.

When a customer arrives at your website, they have a 40% chance of being in the control, 10% of being in A1, 20% of being in B1 and 30% in B2. The MVT system randomly assigns the customer to one of these four options. Once the MVT system assigns customers (using HTTP cookies), they remain assigned to that option until one of two things happen:

The test of the treatment they were assigned to has ended.

OR

If they were in the control group, a new test was added.

Test Types

Merchandising rule test

This test allows you to learn if the merchandising rules that you have created (or are about to create) will cause a negative or positive impact on your KPIs. You can run tests on individual rules, or you can group multiple rules into a single test.

Placement test

This test allows you to turn Algonomy Omnichannel Personalization placements on or off. You can use this test to:

  • Determine if adding a new placement to a page improves the KPIs that you care about

  • See if a placement on the left rail performs better than a placement on the right rail

  • Test recommendation creative designs

Experience Optimizer test

This test lets users using the Experience Optimizer to evaluate how different optimization metrics affect the performance of the placements on a particular page.

RichRelevance On vs Off test

This test compares the performance of the currently configured Algonomy recommendations against no recommendations. It can also be used to measure currently configured Algonomy recommendations against recommendations from a set of simple strategies.

Testing against simple strategies gives you a sense of how your current configuration compares to minimal (not personalized) recommendations. The Algonomy simple strategies test uses only three basic kinds of recommendations:

  • Top-selling products for the site (determined solely by units sold)

  • Top-selling products for a category

  • The most clicked, viewed, and purchased products on the site

You can also use the On vs Off test to compare the performance of Algonomy recommendations against your own solution. Please speak with your Algonomy contact for details.

IMPORTANT: You cannot run an On vs Off test when your site is in Listen Mode.

Preference Center test

This test allows users to test using various options for boost settings.

Engage

This test allows users to decide which page template to use to get profit from optimization of the content show using the rules. Users can test the impact of switching on/off the content show rules in the selected digital channels.